Hybrid models for combination of visual and textual features in context-based image retrieval.

Visual Information Retrieval poses a challenge to intelligent information search systems. This is due to the semantic gap, the difference between human perception (information needs) and the machine representation of multimedia objects. Most existing image retrieval systems are monomodal, as they utilize only visual or only textual information about images. The semantic gap can be reduced by improving existing visual representations, making them suitable for a large-scale generic image retrieval. The best up-to-date candidates for a large-scale Content-based Image Retrieval are models based on the Bag of Visual Words framework. Existing approaches, however, produce high dimensional and thus expensive representations for data storage and computation. Because the standard Bag of Visual Words framework disregards the relationships between the histogram bins, the model can be further enhanced by exploiting the correlations between the visual words. Even the improved visual features will find it hard to capture an abstract semantic meaning of some queries, e.g. straight road in the USA. Textual features, on the other hand, would struggle with such queries as church with more than two towers as in many cases the information about the number of towers would be missing. Thus, both visual and textual features represent complementary yet correlated aspects of the same information object, an image. Existing hybrid approaches for the combination of visual and textual features do not take these inherent relationships into account and thus the combinations performance improvement is limited. Visual and textual features can be also combined in the context of relevance feedback. The relevance feedback can help us narrow down and correct the search. The feedback mechanism would produce subsets of visual query and feedback representations as well as subsets of textual query and textual feedback representations. A meaningful feature combination in the context of relevance feedback should take the inherent inter (visual-textual) and intra (visual-visual, textualtextual) relationships into account. In this work, we propose a principled framework for the semantic gap reduction in large scale generic image retrieval. The proposed framework comprises development and enhancement of novel visual features, a hybrid model for the visual and textual features combination, and a hybrid model for the combination of features in the context of relevance feedback, with both fixed and adaptive weighting schemes (importance of a query and its context). Apart from the experimental evaluation of our models, theoretical validations of some interesting discoveries on feature fusion strategies were also performed. The proposed models were incorporated into our prototype system with an interactive user interface

RGU at ImageCLEF2010 Wikipedia Retrieval Task

This working notes paper describes our first participation in the ImageCLEF2010 Wikipedia Retrieval Task. In this task, we mainly test our Quantum Theory inspired retrieval function on cross media retrieval. Instead of heuristically combining the ranking scores independently from different media types, we develop a tensor product based model to represent textual and visual content features of an image as a non-separable composite system. Such system incorporates the statistical/semantic dependencies between certain features. Then the ranking scores of the images are computed in a way as quantum measurement does. Meanwhile, we also test a new local feature that we have developed for content based image retrieval

Tensor product of correlated text and visual features: a quantum theory inspired image retrieval framework

In multimedia information retrieval, where a document may contain textual and visual content features, the ranking of documents is often computed by heuristically combining the feature spaces of different media types or combining the ranking scores computed independently from different feature spaces. In this paper, we propose a principled approach inspired by Quantum Theory. Specifically, we propose a tensor product based model aiming to represent text and visual content features of an image as a non-separable composite system. The ranking scores of the images are then computed in the form of a quantum measurement. In addition, the correlations between features of different media types are incorporated in the framework. Experiments on ImageClef2007 show a promising performance of the tensor based approach

Improving content based image retrieval by identifying least and most correlated visual words

In this paper, we propose a model for direct incorporation of im- age content into a (short-term) user profile based on correlations between visual words and adaptation of the similarity measure. The relationships between visual words at different contextual levels are explored. We introduce and compare var- ious notions of correlation, which in general we will refer to as image-level and proximity-based. The information about the most and the least correlated visual words can be exploited in order to adapt the similarity measure. The evaluation, preceding an experiment involving real users (future work), is performed within the Pseudo Relevance Feedback framework. We test our new method on three large data collections, namely MIRFlickr, ImageCLEF, and a collection from British National Geological Survey (BGS). The proposed model is computation- ally cheap and scalable to large image collections

Early fusion and query modification in their dual late fusion forms.

In this paper, we prove that specific widely used models in Content-based Image Retrieval for information fusion are interchangeable. In addition, we show that even advanced, non-standard fusion strategies can be represented in dual forms. These models are often classified as representing early or late fusion strategies. We also prove that the standard query modification method with specific similarity measurements can be represented in a late fusion form

Twitter response to televised political debates in Election 2015.

The advent of social media such as Twitter has revolutionised our conversations about live television events. In the days before the Internet, conversation about television programmes was limited to those sitting on the sofa with you and people you met the next morning – so-called ‘water-cooler conversation’. Now, however, it is possible to discuss events on the screen in real time with people all over the country – three out of five UK twitter users tweet while watching television (Nielsen, 2013). Thus it is not surprising to find that the General Election’s television events generated debate and discussion on twitter

Novel local features with hybrid sampling technique for image retrieval

In image retrieval, most existing approaches that incorporate local features produce high dimensional vectors, which lead to a high computational and data storage cost. Moreover, when it comes to the retrieval of generic real-life images, randomly generated patches are often more discriminant than the ones produced by corner/blob detectors. In order to tackle these problems, we propose a novel method incorporating local features with a hybrid sampling (a combination of detector-based and random sampling). We take three large data collections for the evaluation: MIRFlickr, ImageCLEF, and a collection from British National Geological Survey. The overall performance of the proposed approach is better than the performance of global features and comparable with the current state-of-the-art methods in content-based image retrieval. One of the advantages of our method when compared with others is its easy implementation and low computational cost. Another is that hybrid sampling can improve the performance of other methods based on the "bag of visual words" approach

On the duality of specific early and late fusion strategies

In this paper, we prove that specific early and specific late fusion strategies are interchangeable. In the case of the late fusion, we consider not only linear but also nonlinear combinations of scores. Our findings are important from both theoretical and practical (applied) perspectives. The duality of specific fusion strategies also answers the question why in the literature the experimental results for both early and late fusion are often similar. The most important aspect of our research is, however, related to the presumable drawbacks of the aforementioned fusion strategies. It is an accepted fact that the main drawback of the early fusion is the curse of dimensionality (generation of high dimensional vectors) whereas the main drawback of the late fusion is its inability to capture correlation between feature spaces. Our proof on the interchangeability of specific fusion schemes undermines this belief. Only one of the possibilities exists: either the late fusion is capable of capturing the correlation between feature spaces or the interaction between the early fusion operators and the similarity measurements decorrelates feature spaces

Contextual image annotation via projection and quantum theory inspired measurement for integration of text and visual features

Multimedia information retrieval suffers from the semantic gap, a difference between human perception and machine representation of images. In order to reduce the gap, a quantum theory inspired theoretical framework for integration of text and visual features has been proposed. This article is a followup work on this model. Previously, two relatively straight forward statistical approaches for making associations between dimensions of both feature spaces were employed, but with unsatisfactory results. In this paper, we propose to alleviate the problem regarding unannotated images by projecting them onto subspaces representing visual context and by incorporating a quantum-like measurement. The proposed principled approach extends the traditional vector space model (VSM) and seamlessly integrates with the tensor-based framework. Here,we experimentally test the novel association methods in a small-scale experiment